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The Effectiveness of Open Court on Improving the Reading Achievement 
of Economically-Disadvantaged Children 

Margaret Moustafa and Robert Land 
California State University Los Angeles 



This study compares average SAT 9 reading scores of second through 
fifth grade English-only children in schools using Open Court with the scores 
of comparable schools using non-scripted programs in one very large urban 
school district. Findings show that schools using Open Court are significantly 
more likely to be in the bottom quartile than comparable schools using non- 
scripted programs. 

In 1996 the reading / language arts teacher specialists on California’s Instructional 
Resources Evaluation Panel recommended that Open Court’s Collections for Young 
Scholars (1995), a program for teaching children to read, not be placed on the state’s 
textbook adoption list (Holland, 1996). Nevertheless, the California Board of Education 
overruled the Panel and placed it on the California textbook adoption list. By January 
2000, Open Court, as it is commonly referred to, had grown from being used in one in 
every hundred to one in every eight elementary schools in California (Helfand, 2000). 
Since that time it has been adopted in many urban school districts in California and 
around the nation. When it was adopted district-wide in the nation’s second largest school 



district the spring of 2000, it won the largest reading textbook adoption every recorded 
(McGraw-Hill, 2000, p.ll). 

Open Court’s dramatic growth in market share was aided by reports that a 
National Institute of Child Health and Development (NICHD) study, done by a group of 
researchers lead by Barbara Foorman of the University of Texas — Houston Medical 
Center, had found that economically disadvantaged children have better reading 
achievement with Open Court than economically disadvantaged children with whole 
language, or contemporary, reading instruction programs. The research was presented to 
the California State Assembly Education Committee in 1996 and used to promote laws 
before it was peer reviewed (Taylor, 1998). 

Whose views were more valid: those of the reading / language arts teacher 
specialists on the 1996 California Instructional Resources Evaluation panel or those of 
the NICHD Houston researchers? In this paper we analyze the NICHD Houston Study 
and other studies on the effectiveness of Open Court and report on our own study. 

The NICHD Houston Study 

In 1994-95 Foorman, Francis, Fletcher, Schatschneider, & Mehta studied the 
effect of various types of reading programs on the reading achievement of low-achieving, 
economically-disadvantaged, first and second grade children in a school district in Texas. 
The programs they studied were: (1) the pre-publication version of Open Court’s 
Collections for Young Scholars, (2) the researchers’ adaptation of Hiebert, Colt, Catto 
and Gray’s 1992 program, (3) the researchers’ adaptation of contemporary reading 
instruction, and (4) the ongoing contemporary reading instruction in place in the district 
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before the study began. They collected data on children in various classrooms, each with 
one of the four programs. 

Foorman and her colleagues found the children in the classrooms with Open 
Court improved in word reading more than the children in the other classrooms. 
Consequently, they concluded that the “[r]esults show advantages for reading 
instructional programs that emphasize explicit instruction in the alphabetic principle for 
at-risk children” (Foorman, Francis, Fletcher, Schatschneider, & Mehta, 1998, p. 37). 

There are many problems with Foorman et al.’s research. Some of the most 
important problems are: 

1) The version presented to the California State Assembly Education Committee 
May 8, 1996 and the version published after peer review in the Journal of 
Educational Psychology in 1998 used considerably different data (Taylor, 1998, 
333-338). Taylor, Anderson, Au, and Raphael (2000) suggest the changes may 
have been the consequence of peer review. 

2) In both versions, the sample favored Open Court (Taylor, 1998, pp. 333-338; 
Foorman et al., 1998, Table 1). In the published version most of classrooms with 
Open Court were in schools where 40-43% of the children were on the federal 
lunch program. In contrast, most of the classrooms with the adapted Hiebert et al. 
program were in schools where 64-65% of the children were on the federal lunch 
program; most of the classrooms with the adapted contemporary reading 
instruction were in schools where 50-64% of the children were in the federal 
lunch program; and, most of the classrooms with the ongoing contemporary 
instruction were in a school where 71% of the children were on the federal lunch 
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program (Foorman et al., 1998, Table 1). The school that had 71% of the children 
on the federal lunch program also had the lowest achievement scores on the 
statewide test in Grade 3 (Foorman et al., 1998, p. 39). 

3) In both versions, the children receiving Open Court instruction had higher 
average pre-test scores on word reading and phonological processing than the 
children in each of the three comparison groups at both grade levels (Taylor, 

1998, pp. 333-338; Foorman etal., 1998, pp. 43-44). In the prepublication 
version two-tailed t-tests show differences were statistically significant at the .05 
level or less for word reading. In the published version pre-test differences are 
less pronounced, but still favor the Open Court group in every case. 

4) In both versions the data were incorrectly aggregated. The data on the children 
who had been in the district’s classrooms with contemporary reading instruction 
when they were in first grade but in the Open Court classrooms when they were 
in second grade were averaged with the data on the children who were in the 
classrooms with Open Court (Foorman et al., 1998, Tables 3 & 4), making it 
impossible to distinguish the effect of each program. 

5) In both versions the researchers equated pronouncing print words with reading. 
That is, they assumed that if a child can pronounce a print word the child is 
“reading,” regardless of whether the child is making sense of the print or not. 

At the end of the six -month study, despite their lower pre-test scores and their 
greater economic disadvantage, the children in the classrooms with the ongoing 
contemporary reading instruction answered comprehension questions about text they had 




6 



4 



read silently better than the children in the Open Court classrooms (Foorman et al., 1998, 
Table 5). If one defines reading as making sense of print, one would have to conclude 
from the NICHD Houston study that contemporary reading instruction is more effective 
than traditional, parts-to-whole reading instruction. This reanalysis of Foorman et al.’s 
findings is consistent with independent research that has found that all children, but 
especially lower-achieving children, achieve more with contemporary reading instruction 
than with traditional reading instruction (e.g., Anderson, Wilkinson, and Mason, 1991; 
Cantrell, 1999; Eldredge, Reutzel, and Hollingsworth, 1996; Mullis, Campbell, and 
Farstrup, 1993; Reutzel and Cooter, 1990; Sacks and Mergendollar, 1997). 

Despite its flaws, the NICHD Houston research was used to justify policy and 
legislation that excluded contemporary approaches to reading instruction and favored the 
purchase of Open Court, especially in schools serving poor families (Coles, 2000; Taylor, 
1998; Taylor et al., 2000). Teachers who are required to use only Open Court for reading 
instruction consistently report that instruction in writing, math, science, and social studies 
has declined dramatically, if they are taught at all. 

Newspaper Studies 

In June of 1999 Debra Saunders of the San Francisco Chronicle wrote about 
Sacramento City Unified School District’s almost exclusive use of Open Court. She 
reported that since signing on with Open Court and receiving a grant from the David and 
Lucile Packard Foundation for teaching coaches, . .scores for the Sacramento City 
Unified School District rose from the 35 th percentile nationally in reading for first-graders 
[in 1997] to the 54 th percentile [in 1998] to the 62 nd percentile [in 1999]” (Saunders, 
1999). 
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Similarly, in August of the same year, Duke Helfand of the Los Angeles Times 
wrote about Sacramento’s success with Open Court as measured by the SAT 9. He 
reported “The primary grades led the way, with second-graders making the largest gains: 
15 percentile points in reading compared with 4 points among their counterparts across 
the state” (Helfand, 1999, p. Al). 

In their analyses of Open Court's success in Sacramento, Saunders and Helfand 
compared Sacramento’s scores in one grade one year to scores in the same grade the next 
year. However, when we follow the children from one grade one year to the next grade 
the next year we see a different picture. 

If we follow Sacramento’s children from one grade to the next we see that while 
Sacramento’s children averaged in the 35 th percentile in first grade in 1997, they 
averaged in the 35 th percentile in second grade in 1998 and in the 37 th percentile in third 
grade in 1999. While Sacramento’s children averaged in the 54 th percentile in first grade 
in 1998, they averaged in the 50 th percentile in second grade in 1999 and in the 42 nd 
percentile in third grade in 2000. Finally, while Sacramento’s children averaged in the 
62 nd percentile in first grade in 1999, they averaged in the 52 nd percentile in second grade 
in 2000 and in the 43 rd percentile in third grade in 2001. 

Similarly, if we follow California’s children from one grade to the next we see 
that the state’s children averaged in the 39 th percentile in second grade in 1998 and in the 
40 th percentile in 1999. Hence, between 1998 and 1999 children in Sacramento who were 
in second grade in 1998 and in third grade in 1999 gained only one percentile point more 
than the state’s children. Parenthetically, we note that regardless of the fact Sacramento’s 
average second grader scores were higher than the state’s in 1999, 2000, and 2001, this 
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early advantage has never translated into higher scores in subsequent grades. In every 
year SAT 9 had been administered state wide, 1998, 1999, 2000, and 2001, Sacramento’s 
third, fourth, and fifth grade scores have been lower than the state’s. 

The Treadway Study 

After we made a working draft of our study available to others via the web, Jerry 
Treadway (2000), an author of Open Court, did a study on the effectiveness of Open 
Court and made his study available on the University of Texas — Houston, Center for 
Academic and Reading Skills (CARS) website. In his study, Treadway compared the 
average 1999 SAT 9 reading scores of 10 schools that use Open Court with those of 10 
other schools in the same district we studied and concluded that the average SAT 9 
reading scores of the 10 schools using Open Court was higher. However, there are 
methodological flaws in Treadway’s study: 

• Treadway selected 20 of over 400 schools in the district, without explaining his 
selection criteria, making it impossible to know why he chose the particular 20 
schools he chose. 

• The study was not a comparison of Open Court vs. non-scripted programs. District 
records show that two of Tredway’s 10 comparison schools were using Open Court 
and three were using Success for All, another scripted program. 

• The study used meaningless data. It used the SAT 9 reading scores of all students, 
English-only and limited-English proficient, without reference to how long the later 
group had been in English speaking schools. We elaborate on this problem below. 
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Our Study 



Our research asked two questions: 

(1) Does Open Court foster higher reading achievement than non-scripted reading 
programs initially among economically disadvantaged children? 

(2 Does Open Court foster higher reading achievement than non-scripted reading 
programs generally among economically disadvantaged children? 

Traditionally scripted and non-scripted instructional programs are distinguished 
by the amount of professional judgment teachers are allowed to exercise in teaching. 
Traditional scripted programs provide teachers with a script for what they are to say 
verbatim during instruction. Non-scripted programs describe activities, provide examples 
and expect teachers to the choose activities that they judge to be most helpful to 
particular groups of students in their care. 

The teachers’ manual of the 1995 edition of Open Court examined in this study 
looks like that of a non-scripted program. However, in every district-wide adoption we 
are aware of in California, the state in which our study took place, teachers are required 
to complete every activity described in the teachers’ manual with the entire class, whether 
it is appropriate or not, and to do it at a prescribed pace (i.e., so many lessons within so 
many days), whether it is appropriate or not. Michael Bazeley (2000) of the San Jose 
Mercury said “As one of the most strictly orchestrated programs on the market, Open 
Court is helping bury a notion central to public education for decades — that the 
classroom revolves around the teacher as a trained professional in control of what 
happens and when. With Open Court, teachers are told what to do from the minute class 
starts.” Debra Anderluh (1998) of the Sacramento Bee wrote that Open Court “lasts two 
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to three hours a day, with half that time spent in whole group instruction. The detailed 
scripting, combined with the over sight of roving coaches, means that from school to 
school, teachers are presenting basically the same lesson at a given grade level in a given 
week.” We have observed teachers being on the same lesson in a given week from district 
to district. Hence, as currently being implemented, we see the 1995 and 2000 editions of 
Open Court as a new form of scripted instruction. 

Method 

We used the 1999 average SAT 9 reading achievement scores of English-only 
children to measure the success of Open Court vs. non-scripted programs. 

Using SAT 9 scores to compare the success of one reading program with the 
success of other reading programs is problematic. SAT 9 is not a criterion-referenced test 
but a norm-referenced test (NRT). Norm-referenced tests are constructed so that 50% of 
the test takers will score in the bottom 50%, regardless of their mastery of the subject. 
Furthermore, norm referenced tests rarely include items that most children get right 
(Popham, 1999). 

In California where our study took place, another problem in using the SAT 9 to 
assess the effectiveness of reading instruction is that the SAT 9 norms were developed 
with a population where 1.8 percent of the children were limited English proficient. Over 
25% of California’s K-12 children are limited English proficient with higher percentages 
in the lower grades. Hence, schools with large numbers of limited English proficient 
children will compare unfavorably to SAT 9’ s norming group as well as to other 
California schools with lower numbers of limited English proficient children. Moreover, 
the scores of limited English proficient children, as presently reported, are meaningless. 
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Immigration occurs at all ages. It takes a minimum of five to seven years for non-native 
speakers of English to achieve in English at levels equivalent to native speakers (Collier, 
1989). It’s impossible to discern progress — or lack of progress — from data which 
combines the scores of children who have been in U.S. schools a few years with children 
who have been in U.S. schools many years. 

There are other problems with using SAT 9 scores to assess the effectiveness of 
reading programs. SAT 9 scores do not tell us if the teachers at each school are fully 
credential ed or not, if they are teaching the programs as the programs were designed to be 
taught or not, if they are providing instruction above and beyond the programs or not, if 
they are spending significantly more time in reading instruction or not, or if they are 
receiving significantly more assistance from aides, volunteers, or tutors or not. SAT 9 
scores also do not tell us if the children at each school are economically disadvantaged or 
not, if they are the same children that were at that school the year before or not, or if 
significant numbers of children were excluded from taking the test by being placed in 
Special Education or not. 

Nevertheless, given today’s high stakes testing where administrators, teachers and 
school staffs are rewarded or punished on the basis of their SAT 9 scores, an assessment 
of how successful various programs are in enabling students to do well on the SAT 9 
reading assessment may be of interest to educators caught up in high-stakes testing. 

We chose the 1999 SAT 9 reading data because, at the time we began our work, it 
was the first year where the state reported English-only children’s scores disaggregated 
from those of limited English proficient children. Also, 1999 was the second year the SAT 
9 was administered statewide. Presumably, classroom teachers who see the test when 
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they administer it to their students probably had not yet figured out by 1999 that the 
questions on the SAT 9 are the same year after year and would have been less prepared to 
teach to particular test items than they were in subsequent years. 

Working with the 1999 average SAT 9 reading scores of English-only children 
and the percent of children on free / reduced-price meals obtained from the California 
Department of Education’s web site, we investigated the effectiveness of Open Court 
schools against non-scripted programs in schools serving similar economic groups in one 
very large urban school district in California. The non-scripted programs used across 
schools in the district in 1998-99 were Invitations to Literacy (Houghton Mifflin Co.), 
Literacy Places (Scholastic, Inc.), Signatures (Harcourt-Brace), and Spotlight on Literacy 
(McGraw-Hill). 

We limited our study to schools on the traditional calendar in the district. We 
further limited our study to schools that either (1) used Open Court not in combination 
with another program or (2) used one of the non-scripted programs used across the Los 
Angeles Unified School District in 1998-99 not in combination with other programs. 

Finally, our study was limited to second through fifth grade scores. Second grade 
is the earliest grade in which scores are publicly reported and fifth grade is the highest 
elementary grade in the district we studied. 

We found 159 elementary schools in the district that met the research criteria for 
inclusion in the study. Of these, 21 used Open Court. Nine of the Open Court schools had 
been using Open Court over ten years. All the Open Court schools served populations 
where 50% or more of the children were on free / reduced-price meals. Hence, we further 
limited the schools in the study to schools serving populations where 50% or more of the 
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children were on free / reduced price meals. This left 153 schools in the study. Table 1 
shows the number of schools in the study by type of program and economic disadvantage 
of the children. 



Table 1 

Numbers of Schools by 

Type of Reading Program and Percent Children Receiving Free / Reduced-Price Meals 





Children Receiving Free / Reduced-Price Meals 


Reading Program 


50-59% 


60-69% 


70-79% 


80-89% 


90-100% 


Non-scripted programs 


12 


10 


43 


24 


41 


Open Court 


1 


1 


4 


4 


2 


Long-term Open Court 










.. 9 } 



Among 9 of the schools using non-scripted programs with 95-100% of the 
children on free / reduced-price meals, some grades had too few English-only children for 
the scores to be reported by the state. 

To investigate whether Open Court fosters higher reading achievement initially 
among economically disadvantaged children, we compared the average second grade 
scores of the schools in the study. In this comparison we sorted the scores into one of 
three groups: those below the 25 th percentile, those at the 25 th to 49 th percentile, and those 
at the 50 th percentile or higher. Four of the 153 grades in this comparison had too few 
English-only children for the scores to be reported. 

To investigate whether Open Court fosters higher reading achievement generally 
among economically disadvantaged children, we compared the average scores of all the 
grades, second through fifth grade, of long-term Open Court schools with the scores of 
schools using the non-scripted programs that had the same percent of children on free / 
reduced-price meals. The long-term Open Court schools in the district we studied serve 
populations where 97-100% of the children receive free / reduced-price meals. Therefore, 
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we compared the scores of the long-term Open Court schools with the scores of schools 
using non-scrip ted programs where 97-100% of the children received free / reduced-price 
meals. As shown in Table 2, the groups were comparably disadvantaged. The average 
long-term Open Court school in this group had 98.3 children receiving free / reduced- 
price meals. The average school using a non-scripted program had 98.5 children 
receiving free / reduced-price meals. 



Table 2 

Numbers of Schools with 97-100% of the Children on Free / Reduced-Price Meals by 

Type of Reading Program 





Children Receiving Free / Reduced-Price Meals 


Reading Program 


97% 


98% 


99% 


100% 


Non-Scripted Programs 


4 


7 


5 


5 


Long-Term Open Court 


4 


1 


1 


3 



Only one school in the all-grades comparison (a school using a non-scripted 
program) had average scores at 50% or higher. Hence, in this comparison we sorted the 
scores of each grade into one of two groups: those with average SAT 9 reading scores in 
the bottom quartile on the SAT 9 and those with average scores above the bottom quartile. 
Nine of the 120 grades in this comparison had too few English-only children for their 
scores to be reported. 

The fifth-grade score of one long-term Open Court school was omitted in the all- 
grade comparisons because the score was an outlier. While the second grade children in 
this school averaged in the 28 th percentile and the third and fourth grade children 
averaged in the 17 th percentile, the fifth grade children averaged in the 62 nd percentile, a 
difference far greater than in any other school. 
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Findings 



Second grade comparison. Table 3 shows the number of schools with average 
second grade English-only SAT 9 reading scores in the second, third, and fourth quartile. 
The percentage of children in each quartile is about the same for schools using Open 
Court and the non-scripted programs. A 2x3 Chi shows no significant difference in the 
average scores of the schools using Open Court and the average scores of the non- 
scripted programs (p=.545). 



Table 3 

Numbers of Schools with Average Second Grade English-Only SAT 9 Reading Scores 

in the Second, Third, and Fourth Quartile 





Average SAT 9 Reading Scores 


Reading Program 


0-24% 


25-49% 


50% and above 


Non-Scripted Programs 


16(13%) 


94 (73%) 


18(14%) 


Open Court 


4 (19%) 


13 (62%) 


4 (19%) 



Chi 2 (df2) = 1.214, p = .545 



Among just the schools where 97-100% of the children are on free / reduced-price 
meals, the average SAT 9 reading scores for second grade English-only children was 33 
among the 18 schools using non-scripted programs and 31 among the 9 long-term Open 
Court schools. Again, the difference is not statistically significant: t(df25)=-507, p =.617. 

All-grades comparison. As shown in Table 4, in schools where 97-100% of the 
children were receiving free / reduced-price meals, grades in schools using the non- 
scripted programs scored above the bottom quartile almost twice as often as grades in 
long-term Open Court schools. While 72% (54 out of 75) of the grades in the schools 
using non-scripted programs scored above the bottom quartile, only 43% (15 out of 35) 
of the grades in the long-term Open Court schools scored above the bottom quartile. A 
2x2 Chi 2 shows that this difference is statistically significant at the .01 level. 
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Table 4 

Numbers of Grades with Average English-Only SAT 9 Reading Scores 
In and Above the Bottom Quartile 

In Schools Where 97-100% of the Children Receive Free / Reduced-Price Meals 





Average SAT 9 Reading Scores 


Reading Programs 


In bottom quartile 


Above bottom quartile 


Non-Scripted Programs 


21 (28%) 


54 (72%) 


Long Term Open Court 


20 (57%) 


15 (44%) 



Chi 2 (dfl) = 8.669, pc.Ol 



Discussion . 

We found no evidence that Open Court fosters higher early reading achievement 
among economically disadvantaged children. At the second grade level, the earliest grade 
level reported publicly, we found no significant difference in the average SAT 9 reading 
scores between schools using Open Court and schools using the non-scripted programs 
serving similarly disadvantaged children. 

We also found no evidence that Open Court fosters higher reading achievement 
generally among economically disadvantaged children. When we looked at the scores of 
all the elementary grades that are publicly reported (second through fifth grade), we 
found schools that had used Open Court 10 or more years were significantly more likely 
to be in the bottom quartile of the SAT 9 than schools using non-scripted programs 
serving similarly disadvantaged children. 

In other words, we found no support for the assertion that implementing Open 
Court leads to higher SAT 9 scores. We further found no justification in sacrificing 
instruction in other curricular areas to implement Open Court. 

The findings of this study are consistent with the findings of our reanalysis of 
children’s reading in Open Court vs. the contemporary reading instruction in the NICHD 
Houston study in Texas. Altogether, the outcomes in the two school districts suggest that 



Open Court limits what children are able to achieve in literacy relative to what they are 
able to achieve via many other programs. The outcomes support the professional 
judgment of the reading / language arts teacher specialists on California’s 1996 
Instructional Resources Evaluation Panel who recommended Open Court not be placed 
on California’s textbook adoption list. 
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